Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: incremental updates with updatedAfter #71

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Conversation

fracasula
Copy link
Contributor

@fracasula fracasula commented Dec 3, 2024

Description

Adding support for INCREMENTAL UPDATES with updatedAfter + general cleanup.

USAGE SHOWCASE HERE

IMPORTANT DISCLAIMER

Now it is possible to use the SDK and the Poller with any object, not just *modelv2.WorkspaceConfigs.
This is possible as long as the aforementioned object satisfies a few contracts with generics.
In order to be efficient when handling such big objects, iterators have been used (i.e. yield, iter.Seq, iter.Seq2).

Changelog

  • Poller now supports exponential backoff to avoid hammering the ControlPlane when down.
  • Poller isn't internal anymore and it is specific to workspace config changes.
    • it was already specific to that but it wasn't reflected in its name and some of its behaviour.
  • Poller and Diff components expose updated, a variable that allows us to understand if we actually got something new from ControlPlane so that we don't waste computation on the application side rebuilding things every time unless needed.
  • Decoupled components from SDK. Users can now assemble the needed dependencies as they please without having to rely heavily on configuration.
  • Removed unused code (subscribers, cache...).
    • WARNING: these features can be reimplemented once people are actually using them or we'll just find ourselves with unncessary coupling given by dead code.
  • Removed custom client methods (e.g. GetCustomWorkspaceConfigs). Now everything is handled by GetWorkspaceConfigs.
  • Introduced new diff package that leverages generics and iterators (i.e. yield) to efficiently handle the payload.
  • Updated sample app to show case the new SDK.
  • Updated Golang version.
  • Updated GolangCI-lint.

TODOs

Update a few deps once this is merged (see open PRs).

Linear Ticket

< Linear_Link >

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@fracasula fracasula changed the title feat: updatedAfter feat: updatedAfter support Dec 5, 2024
@fracasula fracasula changed the title feat: updatedAfter support feat: incremental updates with updatedAfter Dec 5, 2024
fracasula and others added 3 commits December 6, 2024 17:54
* chore: experimenting with new approach

* fix: 2nd yield

* fix: unused code

* test: new approach

* test: non updateables

* chore: finishing up
@fracasula fracasula marked this pull request as ready for review December 10, 2024 16:50
@ktgowtham
Copy link

@fxenik , could you review it from CP side?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants